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Addendum: 

Accreditation  Recommendation,  10  October  2009 


Each  of  the  MCRS-16  Scenario  Working  Groups  have  reviewed  their  scenario  input  and  output 
data.  Their  data  have  been  provided  to  the  JDS  registry  and  their  Data  V&V  activity  reports  are 
available  from  OSD/CAPE/JDS.  Accreditation  criteria  5,  6,  and  7  have  been  satisfactorily 
completed. 

The  CMARPS  modeling  and  simulation  (M&S)  team  has  completed  its  VV&A  report  for 
MCRS-16.  CMARPS  continues  to  perform  as  advertised  by  its  long  accreditation  history.  The 
MCRS-16  director  should  recommend  accreditation  of  this  model  for  the  uses  of  the  MCRS-16. 

The  Transportation  Command’s  M&S  team  modified  the  AMP  federation  and  its  models 
to  accommodate  the  MCRS-2016  scenarios  through  the  end  of  the  study.  Before  completing  its 
final  runs,  the  M&S  team  completed  sensitivity  testing  to  ensure  that  their  models  remain 
valid.  Test  results  are  available  by  contacting  Dr.  Jay  Marcotte,  at  TRANSCOM  Headquarters, 
jay.marcotte.ctr@ustranscom.mil,  618-220-5141/5133. 

With  the  completion  of  these  items,  the  MCRS-16  decision  maker  can  conclude  that  the 
mobility  and  logistics  M&S  and  their  associated  data  have  successfully  navigated  a  demanding 
and  rigorous  set  of  VV&A  processes  designed  to  ensure  that  they  are  useful  for  addressing  the 
issues  raised  in  MCRS-16. 
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Preface 


This  paper  was  prepared  under  the  task  order  “Analysis  Community  Verification,  Validation 
and  Accreditation  Use  Case  (VV&A  Use  Case).”  This  task  order  is  for  work  to  be  performed 
by  the  Institute  for  Defense  Analyses  (IDA)  under  Contract  W91WAW-09-C-0003  for  the 
Director,  Program  Analysis  and  Evaluation  and  the  Modeling  and  Simulation  Coordination 
Office  (MSCO)  Office  of  the  Deputy  Under  Secretary  of  Defense  (Science  and  Technology). 
The  objective  of  this  task  order  is  to  support  the  development,  sufficiency  review,  and 
acceptance  of  an  Analysis  Community  VV&A  Use  Case  including  a  draft  Department  of 
Defense  VV&A  guideline  for  studies  manual  ready  for  Department-level  review.  This  effort 
was  performed  in  close  concert  with  ongoing  Department-level  studies  projected  for 
completion  in  the  Summer  and  Fall  of  2009.  This  paper  reviews  the  MCRS-16  study  efforts  to 
implement  proposed  DoD  VV&A  guidelines.  The  guidelines  developed  by  this  pilot  effort  will 
be  used  by  other  M&S  Communities  to  develop  their  own  VV&A  guidelines. 

The  author  would  like  to  thank  the  IDA  staff  who  reviewed  this  paper  and  thereby 
significantly  improved  the  product:  Dr.  Stuart  Starr,  Dr.  Thomas  Allen,  Dr.  Geoff  Koretsky, 
and  Mr.  Jim  Kurtz. 
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MCRS-16  Accreditation  Report  Summary 


Executive  Summary 

The  effort  described  in  this  document  is  part  of  a  proof  of  concept  application  of  a  new  set  of 
Validation,  Verification,  and  Accreditation  (VV&A)  guidelines  for  analytic  models  and  simu¬ 
lations  developed  for  OSD/CAPE  by  the  Institute  of  Defense  Analyses  (IDA).  The  guidelines 
were  developed  in  response  to  Government  Accountability  Office  (GAO)  reports  indicating 
inadequate  Department  of  Defense  (DoD)  policy  guidance  for  VV&A.1  In  concert  with  CAPE 
and  building  upon  extensive  information  available  through  subject  matter  experts,  a  set  of 
eight  accreditation  criteria  (see  pages  5-6)  based  on  the  guidelines  were  developed  and  applied 
to  the  mobility/  logistics  models  and  simulations  used  in  the  production  of  results  for  the 
Mobility  Capabilities  and  Requirements  Study  2016  (MCRS-16).  The  case  study  demonstrated 
the  viability  of  the  guidelines  and,  based  on  their  implementation,  suggests  that  when  the 
accreditation  criteria  are  completely  satisfied,  the  mobility/logistics  models  and  simulations 
should  be  accredited  for  use  in  the  MCRS-16  study. 

Introduction 

The  MCRS-16  study  is  a  joint,  collaborative  interagency  study  to  assess  the  Joint  Deployment 
Distribution  Enterprise  (JDDE)  in  support  of  the  National  Security  Strategy  as  executed  in  the 
2016  time  frame.  MCRS-16  supports  the  US  Department  of  Defense’s  (DoD)  Analytic  Agenda 
by  providing  mobility  and  sustaimnent  analyses  used  in  Analytic  Baselines.  The  objective  of  the 
MCRS-16  Accreditation  Plan  (in  Volume  2)  is  to  certify  that  all  Models  and  Simulations  (M&S) 
employed  in  MCRS-16  are  acceptable  to  support  mobility  analysis  within  the  Analytic  Agenda 
and  National  Security  Strategy.  Study  organization,  analytical  methods,  management,  and 
oversight  are  discussed  in  detail  in  the  MCRS-16  Terms  of  Reference  and  Study  Plan,  which  will 
likely  be  released  in  2010  as  part  of  the  MCRS-16  Final  Report. 

This  Accreditation  Report  addresses  the  primary  VV&A  efforts  used  to  provide  a  basis 
for  accrediting  the  mobility  models,  simulations,  and  associated  data  used  for  the  MCRS-16.  It 
covers  the  Analysis  of  Mobility  Platform  (AMP)  Federation  (which  includes  the  Model  for 
Intertheater  Deployment  by  Air  and  Sea  [MIDAS],  the  Enhanced  Logistics  Intra-theater 


1  “Study  Limitations  Raise  Questions  about  the  Adequacy  and  Completeness  of  the  Mobility  Capabilities  Study 
Report,”  US  Government  Accountability  Office,  GAO-06-938,  September  2006;  and  “Issues  Concerning 
Airlift  and  Tanker  Programs,”  US  Government  Accountability  Office,  GAO-07-566T,  7  March  2007. 
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Suppot  Tool  [ELIST],  and  AMP-Port  Analysis  Tool  (AMP -PAT)),  the  Integrated  Computer¬ 
ized  Deployment  System  ICODES,  the  Combined  Mating  and  Ranging  Planning  System 
v.  12. 16.8  (CMARPS),  and  the  Air  Refueling  Combat  Employment  Model  v.1.0  (ARCEM). 
Though  this  document  addresses  all  mobility  models  used  to  support  MCRS-16,  it  does  not 
cover  the  VV&A  for  combat  models  associated  with  Analytic  Baselines  supported  by  MCRS- 
16.  Those  models  will  be  addressed  separately  by  proponent  organizations,  specifically  by  the 
OSD/CAPE  Simulation  and  Analysis  Center  (SAC)  and  the  Joint  Staff,  J-8. 

The  models  in  MCRS-16  simulate  air/land/sea  deployment  to  theater;  sustainment  and 
employment  within  the  theater  area  of  operations  (AOR);  and  air  refueling  operations  en  route 
and  within  the  AOR  (see  Figure  1).  Within  the  AOR,  MCRS-16  uses  a  combination  of 
mobility/logistics  models  and  warfighting  models  to  develop  the  requirements  for  combat 
operations.  Effectively,  the  mobility  models  drop  off  the  assets  at  Air  Ports  of  Debarkation 
(APODs)  and  Sea  Ports  of  Debarkation  (SPODs)  in  theater.  Units  move  into  position  for  the 
warfight,  while  support  material  (e.g.,  food,  water,  petroleum,  oil  and  lubricants,  ammo,  spare 
parts)  are  stockpiled  at  the  APODs,  SPODs,  and  Tactical  Assembly  Areas  (TAAs) 
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Figure  1.  MCRS-16,  Models  and  Simulation  Overview 
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The  AMP  federation  was  used  to  track  sustainment  materiel  by  class  and  on-hand  Days 
of  Supply  (DoS).  AMP  explicitly  moved  sustainment  materiel  to  stock  locations  where  they 
were  consumed  by  local  forces  based  on  Logistics  Factor  Files  (LFF)  for  the  ongoing  activity 
(e.g.,  attack,  defense,  patrol). 

Because  the  warfighting  models  do  not  explicitly  model  the  movement  of  logistics  assets 
within  theater,  the  warfighting  models  are  run  using  the  support  materiel  as  necessary  to  support 
combat  operations.  The  warfighting  models  assume  perfect  logistics  and  perfect  coimnand  and 
control  (C2).  These  assumptions  mean  that  if  the  required  materiel  exists  it  is  assumed  to  be 
moved  to  where  it  is  needed  just  in  time  to  be  used.  As  these  models  run,  an  accounting  is  kept 
of  the  materiel  used  and  where  and  when  it  is  used.  After  the  warfighting  model  runs  are 
complete,  the  mobility/logistics  models  are  re-run  to  ensure  the  JDDE  system  can  meet  the 
materiel  distribution  schedule. 

This  manner  of  connecting  the  mobility/logistics  models  to  the  warfight  models  has  two 
problems  that  can  be  dealt  with  by  the  Study  Director  through  sensitivity  analysis.  The  first 
problem  is  the  assumption  of  “perfect”  C2.  Because  no  operation  ever  attains  perfect  C2,  the 
mobility/logistics  models  were  run  parametrically,  ensuring  an  adequate  buffer  exists  for  less- 
than-perfect  C2.  This  was  handled  in  the  analysis  by  examining  the  DoS  available  to  combat 
and  supporting  units.  The  second  problem  has  a  similar  solution.  Note,  that  the  preceding 
approach  does  not  address  Red  attacks  on  Blue  supply  lines,  which  might  interdict  the 
movement  of  material  or  the  stock  of  supplies  on  hand.  Because  no  operation  goes  forward 
with  zero  losses,  sensitivity  analyses  were  performed  in  a  similar  parametric  manner  to 
examine  the  DoS  needed  to  be  maintained  if  specific  percentage  losses  were  assumed  to  be 
associated  with  Red  activity. 

Sensitivity  analysis  is  an  appropriate  way  to  address  previous  objections  to  the  novel 
approach  connecting  mobility  and  warfighting  models  and  their  outcomes  used  in  MCRS-16 
and  in  the  predecessor  MCS  2005  study. 

VV&A  Guidelines,  the  Accreditation  Plan,  and  Accreditation  Criteria 

After  an  extensive  review  of  the  VV&A  literature,  interviews  of  over  100  key  personnel  from 
the  military  operations  research  community,  and  reviews  of  other  large,  campaign-level 
modeling  and  simulation  efforts  (principally  MCS  2005,  JAS,  STORM,  and  JICM),  guidelines 
was  developed  for  the  Resource  Working  Groups  in  MCRS-16.  The  guidelines  were  centered 
on  basic  steps  that  help  demonstrate  validity  in  a  scientific  context.  They  included: 
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1.  Identifying  the  decisionmaker’s  issues  and  the  M&S  requirements  to  support 
addressing  them. 

2.  Validating  the  conceptual  model  for  each  M&S. 

3.  Verifying  that  the  software  code  for  each  M&S  captures  their  conceptual  model. 

4.  Verifying  and  validating  that  the  data  used  in  each  M&S  are  the  best  available  for  the 
decisionmaker’s  purposes. 

5.  Validating  the  output  from  each  M&S. 

6.  Identifying  assumptions,  limitations,  and  constraints  for  each  M&S  and  assessing  the 
risk  associated  with  using  each  M&S  to  support  the  decision-maker. 

The  Resource  Working  Groups  were  briefed  on  those  guidelines  and  the  VV&A  Working 
Group  provided  continual  assistance  as  they  worked  to  comply  with  the  guidelines. 

At  the  outset,  the  guidelines  required  the  Resource  Working  Groups  to  perfonn  a  pre¬ 
liminary  review  of  the  modeling  and  simulation  requirements  for  MCRS-16  and  its  Essential 
Elements  of  Analysis  (EEAs).  They  ensured  that  their  preliminary  selection  of  M&S  for 
possible  use  would  support  MCRS-16  objectives  and  EEAs.  MCRS-16  had  an  ambitious 
timeline  and  set  of  objectives.  Preliminary  reports  for  each  M&S  were  developed  and  reviewed 
to  ensure  each  M&S  could  contribute  to  study  objectives  in  a  timely  manner.  The  mobility 
M&S  discussed  here  satisfied  the  preliminary  review. 

Based  on  the  VV&A  guidelines,  an  Accreditation  Plan  (included  as  Appendix  A)  was 
developed  for  MCRS-16,  and  all  mobility/logistics  M&S  were  subject  to  its  requirements.  As 
part  of  the  Accreditation  Plan,  MCRS-16  personnel  developed  a  set  of  accreditation  criteria  by 
which  MCRS-16  M&S  could  be  considered  acceptable  for  use  in  MCRS-16;  they  included: 

1.  Suitability  -  The  M&S  is  suitable  to  analyze  the  intended  EEAs  as  identified  in  the 
MCRS-16  Study  Plan.  The  simulations  used  were  identified  and  included  a  description 
of  the  model  and  datasets  including  changes,  updates,  and  upgrades.  The  Resource 
Working  Group  Leads  explained  any  assumptions,  limitations,  or  constraints. 

2.  Validated  by  Subject  Matter  Experts  (SMEs)  -  The  M&S  and  datasets  were  validated 
by  SMEs  for  the  intended  use.  SMEs  reviewed  logic,  flow,  decision  trees,  heuristics, 
and  other  diagrams  as  necessary  to  ensure  the  M&S  and  data  fulfill  a  valid  conceptual 
design  and  represent  mobility  assets  and  operations. 
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3.  Verified  by  SMEs  -  The  M&S  and  datasets  were  subjected  to  a  sensitivity  design  of 
experiments  based  on  EEAs.  SMEs  viewed  the  results  to  ensure  the  M&S  responded 
to  inputs  and  changes  as  expected.  If  results  were  not  within  expectations,  SMEs  and 
M&S  proponents  reassessed  the  capability  for  the  M&S  to  address  EEAs  and  provided 
recommendations  to  the  Study  Director. 

4.  Stable  within  Operating  Environment  -  The  M&S  were  confirmed  to  be  stable  within 
the  operating  domain  of  the  study  and  did  not  require  excessive  artificial  adjustments 
to  successfully  represent  the  MCRS-16  environments  identified  in  the  scenarios 
analyzed  by  the  assigned  EEAs. 

5.  Best  Available  Data  -  The  M&S  will  use  the  best  available  data  validated  and  accredit- 
ted  by  Study  participants.  All  data  shall  be  provided  to  the  Joint  Data  Support  with  their 
accreditation  pedigrees  and  appropriate  meta-data  to  enable  repeatability  and  review. 

6.  Valid  within  Scenario  Environment  -  The  appropriate  Scenario  Working  Group  Leads 
will  review  the  M&S  output  to  ensure  the  M&S  have  properly  incorporated  all  condi¬ 
tions,  data  changes,  environmental  considerations,  and  other  factors  that  would  affect 
results  and  subsequent  analyses. 

7.  Repeatable  -  The  M&S  will  be  repeatable  by  any  authorized  organization  (this  means 
an  authorized  organization  will  be  able  to  obtain  the  exact  version  of  the  M&S  and  its 
associated  data  and  reproduce  the  same  Scenario  results,  ostensibly  as  a  starting  point 
for  their  own  analytical  purposes).  All  data  and  conditions  of  operations  will  be 
documented  to  enable  independent  operations.  All  modeling  record  runs  will  be 
archived  with  the  appropriate  versions  of  the  model’s  executable  code. 

8.  Valid  Federation  or  Confederation  -  All  M&S  including  federations  will  be  examined 
holistically  to  ensure  simulation  and  dataset  interactions  are  understood  relative  to 
analytical  considerations.  Each  element  of  the  federation  and/or  confederation  as  well 
as  the  whole  must  be  accredited  to  meet  the  standards  described  above. 

Based  on  the  Accreditation  Plan,  the  Resource  Working  Groups  and  their  M&S  teams 
developed  their  own  V&V  plans  and  proceeded  to  V&V  their  M&S  and  data.  Their  V&V 
reports  and  Accreditation  reports  discuss  at  length  how  each  satisfied  the  eight  criteria  above. 
Current  copies  of  the  V&V  and  Accreditation  reports  (in  most  cases  drafts)  accompany  this 
report  on  a  separate  CD. 

One  model,  ICODES,  was  judged  to  have  already  demonstrated  a  VV&A  history  that 
satisfied  the  OSD  VV&A  guidelines.  Consequently,  ICODES  was  judged  to  be  accredited  for 
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the  MCRS-16  study.  The  documentation  demonstrating  their  adequacy  is  included  on  the 
separate  CD. 

Reviews 

MCRS-16  incorporated  an  extensive  review  process,  including: 

■  A  General  Officer  Executive  Committee  (EXCOM)  and  General  Officer  steering 
committee  (GOSC)  have  met  several  times  to  review  the  MCRS-16  results  and 
monitor  the  study  progress. 

■  Council  of  Colonels  and  Captains  (0-6)  Board  Review.  Approximately  50  military 
and  civilian  personnel  from  across  DoD  conducted  regular  progress  reviews  of  the 
study.  These  progress  reviews  considered  the  results  each  working  group  produced 
(e.g.,  Resources,  Log  CONOPS,  Scenarios)  throughout  the  study.  Reviews  at  this 
level  were  extensive.  Issues  were  consistently  raised  and  dealt  with;  for  a  detailed 
examination  of  their  extent  contact  Mr.  Steve  Ross  of  OSD/CAPE/JDS  for  his  notes  of 
these  meetings.  In  particular,  they  reviewed  the  results  from  each  Scenario  Working 
Group  and  the  outcomes  of  the  Internal  Review  Boards  for  each  simulation 
undergoing  the  VV&A  processes. 

■  Internal  Review  Boards.  The  VV&A  Working  Group  conducted  internal  reviews  of 
each  of  the  mobility  simulations  used  in  the  study.  Experts  were  drawn  from  outside 
the  Resource  Working  Groups  to  review  the  W&A  efforts  and  the  M&S  efforts  of 
the  Resource  Working  Groups.  Preliminary  M&S  reports,  M&S  documentation, 
configuration  management  plans,  draft  V&V  reports,  and  draft  Accreditation  reports 
were  reviewed  by  members  of  this  body. 

■  The  reviews  covered  a  range  of  issues  including  M&S  conceptual  models,  algorithms, 
datasets,  input  data,  output  results,  sensitivity  analyses,  parameter  settings,  logistics, 
concepts  of  operations  and  how  they  were  modeled,  M&S  limitations,  M&S 
constraints,  and  M&S  assumptions. 

■  Each  M&S  was  reviewed  until  the  entire  set  of  board  members  was  satisfied  that  all 
their  questions  were  answered  satisfactorily,  then  the  M&S  was  deemed  to  have 
satisfactorily  completed  the  Internal  Review.  These  reviews  were  conducted  during  a 
period  of  four  months  and  encompassed  five  days  of  questions  and  answers.  The 
members  of  these  internal  reviews  included: 

D  Dr.  Jack  Jackson,  Institute  for  Defense  Analyses,  campaign  analysis  and  combat 
modeling  expert 
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D  CDR  Phil  Poumelle,  OSD/PA&E,  campaign  analysis  modeling  and  maritime 
transportation  expert 

D  Dr.  Geoff  Koretsky,  Institute  for  Defense  Analyses,  mobility  modeling  expert 
D  Ms.  Cherie  Emerson,  HQ/US  Army,  ground  transportation  expert 
D  Mr.  David  Lyle,  HQ/Transportation  Command,  maritime  transportation  expert 
D  LTC  Ty  Prevett,  Air  Force/A9,  air  mobility  and  air  refueling  modeling  and  air 
transportation  expert 

D  Mr.  Ray  Miller,  Air  Force/A9C,  logistics  modeling  expert 

D  Maj  Elizabeth  Hanson,  Air  Force/A9,  air  mobility  modeling  and  transportation  expert 
D  In  addition,  many  of  the  reviews  were  attended  by  Mr.  Dave  Merrill,  Air 
Force/AMC/A9  and  Col  Jean  Mahan,  Deputy  MCRS-16  Study  Director. 

■  Scenario  Working  Group  reviews.  The  input  data  for  each  scenario  were  reviewed  by 
each  individual  Scenario  Working  Group  in  conjunction  with  functional  working  groups 
as  appropriate.  For  example,  the  CONOPS  Working  Groups  reviewed  the  concept  of 
operations  for  each  Scenario  Working  Group.  Working  group  membership  was  selected 
to  be  inclusive  of  each  Service  when  appropriate.  As  results  were  produced  from  each 
model,  the  Scenario  Working  Groups  reviewed  and  validated  them.  Anomalies  were 
brought  to  the  attention  of  the  Study  Director  who  along  with  his  Council  of  Colonels 
assessed  the  model  adequacy  or  the  requirement  for  further  sensitivity  analysis. 

■  Resource  Working  Group  (M&S  Team)  Evaluations.  Each  M&S  analyst  team  was 
responsible  for  verifying  and  validating  their  particular  simulation  (including  their 
conceptual  models,  configuration  management  plans,  computer  software,  and  simula¬ 
tion  results).  Each  team  had  to  submit  VV&A  documentation  to  the  VV&A  Working 
Group  and  Study  Director  for  review  before  an  accreditation  recommendation.  Within 
this  documentation  each  M&S  team  identified  any  M&S  assumptions,  limitations,  and 
constraints  that  could  affect  MCRS-16.  Each  team  produced  products  to  support  the 
Working  Group  reviews,  the  Internal  Review  Boards,  and  the  0-6  Council  Review. 

Each  of  the  M&S  completed  the  Internal  Review  Boards  required  by  the  VV&A  Working 
Group.  Each  of  the  M&S  has  successfully  negotiated  the  hurdles  with  each  succeeding  review. 
Though  no  known  problems  exist  with  completion,  not  all  of  the  reviews  are  complete.  In 
particular,  as  the  study  time  frame  has  been  extended,  not  all  of  the  Scenarios  Working  Groups 
have  completed  their  output  data  reviews.  Because  these  reviews  are  not  complete,  the  M&S 
and  their  data  have  also  not  yet  been  deposited  into  the  Joint  Data  System  (JDS)  registry. 
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Extensive  reviews  have  been  conducted  of  the  MCRS-16  mobility  M&S,  their  associated 
data,  and  their  output  results.  Members  of  the  Internal  Review  Board  are  unanimous  in 
recommending  that  the  mobility/logistics  M&S  be  accredited  for  MCRS-16. 

Risk  Assessment 

Included  as  a  classified  Volume  2  of  this  report  are  the  current  draft  V&V  and  Accreditation 
reports  for  each  M&S  used  in  the  MCRS-16  study.  Within  those  reports,  readers  will  find 
listed  each  M&S’s  assumptions,  limitations  and  any  constraints  on  their  usage.  Each  M&S’s 
Accreditation  report  also  contains  a  risk  assessment  made  based  on  usage  in  MCRS-16. 

A  few  additional  comments  deserve  mentioning: 

■  Several  of  the  models  are  in  daily  operational  use  by  current  operational  planners. 
ICODES  is  used  worldwide  to  manage  and  load  maritime  cargo.  CMARPS  has  been 
in  use  since  the  early  1990s  to  plan  daily  air  refueling  missions  and  allocate  scarce 
tanker  assets  to  operational  requirements.  Both  were  accredited  for  their  usage  outside 
this  study;  ICODES  by  the  Lloyd’s  Registry  of  London,  United  Kingdom,  and 
CMARPS  by  AF/AMC/A9  for  use  in  several  previous  tanker  studies.  In  2007,  both 
the  United  States  Government  Accounting  Office  and  the  USAF  Inspector  General 
cited  CMARPS  for  consistent  VV&A  practices  during  the  Air  Force  KC-X  study. 
Both  models  daily  produce  results  that  are  measured  and  judged  against  actual  results 
in  the  field. 

■  The  other  air  refueling  model  used  in  MCRS-16,  ARCEM,  is  regularly  run  against 
CMARPS  using  similar  simulated  conditions.  Though  not  done  for  all  scenarios  and 
situations  within  the  MCRS-16,  ARCEM  has  consistently  compared  favorably  and 
regularly  matches  CMARPS  results. 

■  ARCEM  and  the  AMP  federation,  as  well  as  the  set  of  underlying  models  (AMP-PAT, 
MIDAS,  and  ELIST)  which  compose  the  federation,  were  stress-tested  to  examine  the 
maximum  deviations  from  operational  accuracy  that  might  be  expected  through  use  of 
the  mobility/logistics  models.  The  expected  accuracy  consistently  falls  within  1-2% 
for  key  MCRS-16  metrics. 

New  OSD  VV&A  Guidelines  will  include  a  risk  assessment  scale  based  on  the  work  of 
Clemence  and  Hartley,  et  al.2  The  risk  assessment  scale  seeks  to  measure  the  risk  associated 
within  the  conceptual  model  and  data  validation  and  ranges  from  zero  to  five: 


2  From  Table  1,  VV&A  Filial  Report  for  PMESII  Models,  by  Evidenced  Based  Research  Inc.,  July  2007. 
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5  -  expresses  fully  validated  theory,  e.g.,  Newtonian  physics 

4  -  expresses  well-researched  theory  with  considerable  data  checks  and  peer 
review,  e.g.,  published  in  textbooks 

3  -  expresses  theory  supported  by  data  and  peer  reviewed  literature,  e.g.,  published 
in  refereed  journals 

2  -  expresses  theory  and  data  with  rational  basis  accepted  by  SMEs  as  plausible, 
vetted  in  open  forums,  conferences,  etc. 

1  -  expresses  codified  theory 

0  -  uncodified  theory  with  mental  model  of  uncertain  consistency  and  completeness 

Campaign-level  models  typically  expect  to  be  assessed  between  levels  0  and  2  on  this 
scale.  In  fact,  some  M&S  used  in  MCRS-16  meet  a  higher  standard  on  this  scale.  ICODES  and 
CMARPS  would  rate  a  4  on  this  scale  because  they  are  in  daily  operational  and  war  planning  use 
and  express  well-researched  theory  with  considerable  data  checks  and  peer  review.  For  MCRS- 
16,  the  M&S  team  using  ARCEM  completely  rescrubbed  all  internal  data  with  USAF/AMC/ 
Stan  Eval.  AMC/Stan  Eval  are  the  USAF  air-refueling  data  experts;  their  certifycation  of  the 
ARCEM  data  is  contained  in  the  ARCEM  V&V  and  Accreditation  Reports.  ARCEM  is  often 
compared  with  CMARPS  and  would  score  only  slightly  below  CMARPS.  The  AMP  federation 
and  its  sub-models  rate  a  risk  assessment  of  2,  as  they  represent  rational  theory  and  data.  Their 
theory,  data,  and  results  are  regularly  vetted  in  open  forums  and  conferences. 

Accreditation  Recommendation 

Each  of  the  MCRS-16  mobility/logistics  models  has  a  documented  conceptual  model  and  an 
active  configuration  management  plan.  Each  mobility  model  has  satisfied  the  eight  accreditation 
criteria,  with  the  exception  that  a  few  of  the  Scenario  Working  Groups  have  not  completed  their 
Scenario  reviews  of  output  data;  their  data  have  not  been  provide  to  JDS  registry,  and  therefore 
has  not  been  posted  to  the  JDS  website.  Consequently,  accreditation  criteria  5,  6,  and  7  have  not 
yet  been  satisfied.  They  should  be  completed  with  dispatch  as  the  study  is  completed. 

Draft  V&V  reports  have  been  prepared  and  reviewed  for  each  of  the  mobility/logistics 
models.  Draft  Accreditation  reports  have  been  prepared  for  the  AMP  federation,  ICODES,  and 
ARCEM.  Each  of  the  draft  reports  has  been  reviewed  and  minor  comments  provided  to  the 
authors. 

CMARPS  has  a  long  VV&A  history  and  though  no  Accreditation  Report  has  been 
produced  yet,  no  problems  are  expected  by  their  modeling  and  simulation  team  in  completing 
their  activities  and  reports  in  a  timely  manner  with  the  MCRS-16  study  completion. 
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The  Transportation  Command  M&S  team  has  continued  to  modify  the  AMP  federation  and 
its  models  to  accommodate  the  MCRS-16  scenarios.  They  expect  when  final  scenario  results  are 
produced  that  they  will  include  complete  sensitivity  testing  to  ensure  their  models  remain  valid. 
They  will  then  complete  the  final  V&V  and  Accreditation  reports  for  the  MCRS-16  Study 
Director  review. 

With  completion  of  the  items  above,  the  MCRS-16  decision-maker  can  conclude  that  the 
mobility  and  logistics  M&S  and  their  associated  data  have  completed  a  demanding  and 
rigorous  set  of  VV&A  processes  designed  to  ensure  the  M&S  and  their  data  are  useful  for 
addressing  the  issues  raised  in  the  MCRS  study. 

With  the  completion  and  review  of  the  items  mentioned  above,  the  Study  Director  should 
recommend  the  MCRS-16  mobility  models  and  their  associated  data  be  accredited  by  the 
Study  Sponsor,  the  Deputy  Secretary  of  Defense. 
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